Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: general value extractor #95

Merged
merged 7 commits into from
Oct 18, 2024
Merged

Conversation

lonerapier
Copy link
Collaborator

@lonerapier lonerapier commented Oct 8, 2024

closes #94 #67

  • adds a general serde-like Value extractor from arbitrary JSON
  • Adds 2 different templates for object and array

Logic is similar to before:

  • check key matches
  • check inside value
  • propogate matched key to value
  • create mask
  • create a subarray from mask

@lonerapier lonerapier changed the title Feat/general value extractor feat: general value extractor Oct 8, 2024
Copy link
Contributor

@Autoparallel Autoparallel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on this: Do we want to have a bit more logical separation that is conducive to NIVC?

For example, perhaps we can parse the JSON into the machine. Pass the machine as step_out into interpreter stage that grabs the location (indices) of where the intended object is and then pass on this new subarray of the machine into the next iteration of this process.

For example, consider:

{
   "a": {
       "inner_a": { 
           "inner_inner_a": "my_string_value"
        }
    }
}

where we want to extract a.inner_a.inner_inner_a. We parse the whole JSON, pass machine into ExtractValue which outputs data of the start and end of a (and here we can even trim bottom off stack and truncate the machine state further possibly, though let's not get into pre-mature optimization), then this is passed again into ExtractValue, and it returns start and end indices of inner_a as step_out. Finally, we know we are at the final retrieval so we call a new circuit instance in NIVC ExtractStringValue which then hunts within the range output previously for inner_inner_a and gets the String value there.

I imagine the role of array is similar and we can just choose to use ExtractArray in NIVC

@lonerapier lonerapier linked an issue Oct 11, 2024 that may be closed by this pull request
8 tasks
@Autoparallel
Copy link
Contributor

My latest commit just broke a ton of tests, however, many of those are codegen tests.

@Autoparallel
Copy link
Contributor

Alright I think this will soon be merged.

Ultimately I want to bring in the NIVC circuits so we can test them properly in here (where life is easy).

@Autoparallel Autoparallel merged commit af08a7a into main Oct 18, 2024
4 checks passed
@Autoparallel Autoparallel deleted the feat/general-value-extractor branch October 18, 2024 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat(extractor): general serde-based value extractor feat(extractor): improvements
2 participants